Information processing device and information processing method

ABSTRACT

An information processing apparatus may include a user recognition unit to recognize a user in a captured image, and a behavior recognition unit to recognize a behavior of a user. In addition, the apparatus may include a generation unit to generate user behavior information including information of the recognized user and the recognized behavior of the recognized user. Further, the apparatus may include a communication unit to transmit the user behavior information to an external apparatus.

CROSS-REFERENCE TO RELATED APPLICATIONS

This present application is a continuation of a national phase entryunder 35 U.S.C. §371 of U.S. application Ser. No. 13/876,580 filed Mar.28, 2013 which claims benefit of International Application No.PCT/JP2011/005535 filed Sep. 30, 2011, which claims benefit to JapanesePatent Application No. 2010-227868 filed in the Japan Patent Office onOct. 7, 2010, the entire contents of which are hereby incorporated byreference.

TECHNICAL FIELD

The present disclosure relates to an information processing device andan information processing method.

BACKGROUND ART

Telephones, so-called videophones, video conferencing systems and thelike are used as devices for interaction between users at remotelocations. Further, a method of performing text chat or video chat withvideo and speech through Internet connection using personal computersand the like is also used.

In such a background, devices such as electronic equipment are becomingmore sophisticated and multifunctional, and, a television receiver, forexample, has a network communication function so that it can not onlyreceive and display video and speech contents of a program from abroadcasting station but also exchange various kinds of information withanother receiver.

For example, JP 2006-50370A discloses a technique that, when a userviews a program content of television broadcasting with a televisionreceiver, displays information related to other users registered (e.g.,videos of other users, thumbnail images of other users, names, channelsor videos of contents viewed by other users etc.) concurrently with theprogram content.

SUMMARY Technical Problem

In the technique disclosed in JP 2006-50370A described above, thetelevision receiver receives a video or speech from the otherreceiver(s) in one location or a plurality of locations and plays thereceived video or speech concurrently with the display of the programcontent, for example, to thereby enable the user to share the sameprogram content and communicate face to face with a user at a remotelocation.

However, in the case where there are a plurality of users of the otherreceiver in one location, when any one of the plurality of users of theother receiver performs some act of behavior such as speech, forexample, it is difficult for a user of the television receiver to beaware of which user of the plurality of users of the other receiver hasperformed the behavior from the playback of the video or speech receivedfrom the other receiver.

In light of the foregoing, it is desirable to provide novel and improvedinformation processing device and information processing method thatenable a user to be aware of which user of another informationprocessing device has performed a behavior.

Solution to Problem

In accordance with one embodiment, an information processing apparatusmay include a user recognition unit to recognize a user in a capturedimage, and a behavior recognition unit to recognize a behavior of auser. In addition, the apparatus may include a generation unit togenerate user behavior information including information of therecognized user and the recognized behavior of the recognized user, anda communication unit to transmit the user behavior information to anexternal apparatus.

In accordance with another embodiment, an information processingapparatus may include a communication unit to receive from an externalapparatus user behavior information including information of arecognized user and a recognized behavior of the recognized user, and ageneration unit to control display on a display screen of an imageincluding a user image corresponding to the information of therecognized user and a visual representation corresponding to theinformation of the recognized behavior.

In accordance with another embodiment, a method of informationprocessing may include recognizing a user in a captured image andrecognizing a behavior of a user. Further, the method may includegenerating, by a processor, user behavior information includinginformation of the recognized user and the recognized behavior of therecognized user, and transmitting the user behavior information to anexternal apparatus.

In accordance with another embodiment, a method of informationprocessing may include receiving from an external apparatus userbehavior information including information of a recognized user and arecognized behavior of the recognized user, and controlling, by aprocessor, display on a display screen of an image including a userimage corresponding to the information of the recognized user and avisual representation corresponding to the information of the recognizedbehavior.

In accordance with another embodiment, a non-transitory recording mediummay be recorded with a computer-readable program executable by acomputer. The program may include recognizing a user in a captured imageand recognizing a behavior of a user. The program may further includegenerating user behavior information including information of therecognized user and the recognized behavior of the recognized user, andtransmitting the user behavior information to an external apparatus.

In accordance with another embodiment, a non-transitory recording mediummay be recorded with a computer-readable program executable by acomputer. The program may include receiving from an external apparatususer behavior information including information of a recognized user anda recognized behavior of the recognized user, and controlling display ona display screen of an image including a user image corresponding to theinformation of the recognized user and a visual representationcorresponding to the information of the recognized behavior.

Advantageous Effects of Invention

As described above, the present disclosure enables a user to be aware ofwhich user of another information processing device has performed abehavior.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an explanatory view illustrating an example of a displaysystem according to an embodiment of the disclosure.

FIG. 2 is an explanatory view illustrating an example of a functionalconfiguration of an information processing device according to a firstembodiment of the disclosure.

FIG. 3 is an explanatory view illustrating an example of a hardwareconfiguration of the information processing device in FIG. 2.

FIG. 4 is a flowchart of a display process executed by the informationprocessing device in FIG. 2.

FIG. 5 is a flowchart of user display processing performed in Step S114in the display process in FIG. 4.

FIG. 6A to FIG. 6D are explanatory views illustrating an example 1 ofthe disclosure.

FIG. 7A to FIG. 7D are explanatory views illustrating the example 1 ofthe disclosure.

FIG. 8A to FIG. 8D are explanatory views illustrating an example 2 ofthe disclosure.

FIG. 9A to FIG. 9D are explanatory views illustrating an example 3 ofthe disclosure.

FIG. 10A to FIG. 10B are explanatory views illustrating an example 4 ofthe disclosure.

FIG. 11A to FIG. 11E are explanatory views illustrating an example 5 ofthe disclosure.

FIG. 12A to FIG. 12B are explanatory views illustrating an example 6 ofthe disclosure.

FIG. 13A to FIG. 13L are explanatory views illustrating an example 7 ofthe disclosure.

FIG. 14 is an explanatory view illustrating an example of a functionalconfiguration of an information processing device according to a secondembodiment of the disclosure.

DESCRIPTION OF EMBODIMENTS

Hereinafter, preferred embodiments of the present disclosure will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

Note that the description will be provided in the following order.

1. Display System

2. Functional Configuration of Information Processing Device (FirstEmbodiment)

3. Hardware Configuration of Information Processing Device

4. Display Process

5. Examples

6. Functional Configuration of Information Processing Device (SecondEmbodiment)

1. Display System

A display system according to an embodiment of the disclosure isdescribed firstly. FIG. 1 is an explanatory view illustrating an exampleof a display system according to the embodiment. FIG. 1 is a front viewwhere the display system is viewed from the front.

In FIG. 1, a display system 100 includes a display device 102 and animaging device 104, for example.

The display device 102 can display still images or moving images basedon drive signals. For example, the display device 102 displays stillimages or moving images by liquid crystals. Note that the display device102 may display still images or moving images using a self-luminousdisplay device such as an organic EL (Electro-Luminescence).

The imaging device 104 is placed at the center of the upper part of thedisplay device 102, and it can capture a subject image in the displaydirection of the display device 102. The imaging device 104 may capturea still image or a moving image using a CCD (Charge Coupled Device)image sensor or using a CMOS (Complementary Metal Oxide Semiconductor)image sensor.

Note that, although the imaging device 104 is mounted at the center ofthe upper part of the display device 102 in this embodiment, the placewhere the imaging device 104 is mounted is not limited to the center ofthe upper part of the display device 102. For example, the imagingdevice 104 may be mounted at the center of the lower part of the displaydevice 102. Further, although one imaging device 104 is mounted in thisembodiment, the number of imaging devices 104 is not limited to one. Forexample, two or more imaging devices 104 may be mounted. Furthermore,although the display device 102 and the imaging device 104 areintegrated in a single piece in this embodiment, the display device 102and the imaging device 104 may be separated into different pieces.

Note that, in the display system 100, a sensor (not shown) that detectsthe presence or absence of a user in front of the display device 102, asignal receiving unit (not shown) that is capable of receiving a controlsignal by infrared ray or radio wave from a remote controller (notshown) may be mounted. The control signal is an example of an operationsignal of the disclosure. Further, the sensor may be capable ofdetecting the distance between the display device 102 and a user infront of the display device 102. Furthermore, a sound acquisition meansthat is capable of acquiring the surrounding sound, such as amicrophone, for example, may be mounted in the display system 100.

The display device 102 according to the embodiment can display contentsof still images or moving images, and display images generated by aninformation processing device 200 in a display process of FIG. 4, asdescribed later.

Further, the imaging device 104 according to the embodiment can capturestill images or moving images of users A and B looking at the displayscreen of the display device 102 in FIG. 2.

2. Functional Configuration of Information Processing Device FirstEmbodiment

Next, a functional configuration of an information processing deviceaccording to a first embodiment of the disclosure is described. FIG. 2is an explanatory view illustrating an example of a functionalconfiguration of an information processing device according to theembodiment. Note that FIG. 2 shows the display system 100 that transmitsa captured image to the information processing device 200, which is aninformation processing device according to the embodiment, and receivesa signal for driving the display device 102 from the informationprocessing device 200, and users A and B, which are users of the displaysystem 100 and the information processing device 200. FIG. 2 furthershows a communication network 800 to which the information processingdevice 200 can connect, a communication server 300 and other informationprocessing devices 500 and 700 that can connect to the communicationnetwork 800, another display system 400 that transmits a captured imageto the information processing device 500 and receives a signal from theinformation processing device 500, users C and D, which are users of thedisplay system 400 and the information processing device 500, anotherdisplay system 600 that transmits a captured image to the informationprocessing device 700 and receives a signal from the informationprocessing device 700, and users E and F, which are users of the displaysystem 600 and the information processing device 700. The other displaysystems 400 and 600 have the same configuration as the display system100, and detailed explanation thereof is omitted. Further, the otherinformation processing devices 500 and 700 have the same configurationas the information processing device 200, and detailed explanationthereof is omitted. The other information processing devices 500 and 700are examples of a communication target device in the disclosure.

In FIG. 2, the information processing device 200 includes an image inputunit 202, a face recognition unit 204, a display image generation unit206, an output unit 208, a display control unit 210, a communicationunit 212, a speech input unit 214, a speech recognition unit 216, anoperation signal input unit 218, an action recognition unit 220, and astorage unit 222, for example.

The image input unit 202, which is an example of an input unit in thedisclosure, receives a captured image generated by imaging in theimaging device 104. Then, the image input unit 202 transmits thereceived (input) captured image to the communication unit 212. Note thatthe image input unit 202 may transmit the received captured image to theface recognition unit 204 or the display image generation unit 206.

The face recognition unit 204, which is an example of a recognition unitin the disclosure, receives a captured image from the other informationprocessing device 500, 700 which is received by the communication unit212, detects faces of users of the information processing device 500,700 from the received captured image, and recognizes users of the otherinformation processing device 500, 700, i.e., the users C and D or theusers E and F, based on the detection result of faces. The capturedimage from the other information processing device 500, 700 is anexample of a predetermined signal related to the behavior of users ofthe communication target device according to the disclosure. Forexample, the face recognition unit 204 recognizes users appearing in thecaptured image by detecting a face part of each user from the receivedcapture image, computes information about the user's face (hereinafterreferred to as “user face information”) from the detected face part ofthe user, and comparing the computed user face information with userface information 223 for each user which is previously computed andstored in the storage unit 222. The act of appearing in a captured imageis an example of the behavior in the disclosure. Note that the facerecognition unit 204 may further recognize the gender, age or the likeof users appearing in the captured image. Then, the face recognitionunit 204 transmits a result of recognition of the users of theinformation processing device 500, 700 to the display image generationunit 206. Further, the face recognition unit 204 may recognize users ofthe information processing device 200, which are the users A and B, byreceiving a captured image from the image input unit 202 and recognizingthe faces of users of the information processing device 200 from thereceived captured image. The captured image from the image input unit202 is an example of a predetermined signal related to the behavior ofusers of the information processing device according to the disclosure.The face recognition unit 204 may then transmit a result of recognitionof the users of the information processing device 200 to the displayimage generation unit 206 or the communication unit 212.

Further, the face recognition unit 204 may recognize facial behaviors ofthe users of the other information processing device 500, 700 from thecaptured image from the other information processing device 500, 700.The act of changing facial behaviors is an example of the behavior inthe disclosure. Then, the face recognition unit 204 may transmit aresult of recognition of the facial behaviors of the users of the otherinformation processing device 500, 700 to the display image generationunit 206. Note that the face recognition unit 204 may recognize facialbehaviors of the users of the information processing device 200 from thecaptured image from the image input unit 202. Then, the face recognitionunit 204 may transmit a result of recognition of facial behaviors of theusers of the information processing device 200 to the display imagegeneration unit 206 or the communication unit 212.

The display image generation unit 206, which is an example of a controlunit in the disclosure, receives a result of recognition of the users ofthe other information processing device 500, 700 from the facerecognition unit 204 and further receives a captured image captured inthe other display system 400 or a captured image captured in the otherdisplay system 600 which is received by the communication unit 212.Then, based on the received result of recognition of the users of theother information processing device 500, 700, the display imagegeneration unit 206 generates a display image in which user imagesrepresenting the users of the other information processing device 500,700 are displayed for each user. The detail of the display image isdescribed later. Note that the display image generation unit 206 maygenerate the user images in the display image by extracting the users'facial parts from the received captured image captured in the otherdisplay system 400 or the received captured image captured in the otherdisplay system 600. Further, the display image generation unit 206 mayuse an avatar image representing each user, for example, as the userimages in the display image. Then, the display image generation unit 206transmits the generated display image to the display control unit 210.

Further, when the display image generation unit 206 receives a capturedimage captured in the display system 100 from the image input unit 202,receives a result of recognition of the users of the informationprocessing device 200 and a result of recognition of the users of theother information processing device 500, 700 from the face recognitionunit 204, and receives a captured image captured in the other displaysystem 400 or a captured image captured in the other display system 600which is received by the communication unit 212, the display imagegeneration unit 206 generates a display image in which user imagesrepresenting the users of the information processing device 200 and theother information processing device 500, 700 are displayed for each userbased on the received result of recognition of the users of theinformation processing device 200 and users of the other informationprocessing device 500, 700.

Further, when the communication unit 212 receives a result ofrecognition of users from a captured image captured in the other displaysystem 400 or a result of recognition of users from a captured imagecaptured in the other display system 600, the display image generationunit 206 may receive the result of recognition and generate a displayimage based on the result of recognition received from the communicationunit 212 instead of a result of recognition by the face recognition unit204. In this case, the above-described face recognition unit 204 mayskip the recognition of users from the captured image from the otherinformation processing device 500, 700.

Further, when the display image generation unit 206 further receives aresult of recognition of facial behaviors of the users of the otherinformation processing device 500, 700 from the face recognition unit204, the display image generation unit 206 generates a display image inwhich a user image representing a user whose facial behavior has changedand the facial behavior are displayed in association with each other inthe above-described display image based further on the received resultof recognition of facial behaviors of the users of the other informationprocessing device 500, 700. For example, when the display imagegeneration unit 206 recognizes that a facial behavior of a certain userhas changed to smiling, the display image generation unit 206 generatesa display image in which a user image representing the user is displayedlike jumping.

Further, when the display image generation unit 206 further receives aresult of recognition of facial behaviors of the users of theinformation processing device 200 from the face recognition unit 204,the display image generation unit 206 generates a display image in whicha user image representing a user whose facial behavior has changed andthe facial behavior are displayed in association with each other in theabove-described display image based further on the received result ofrecognition of facial behaviors of the users of the informationprocessing device 200.

Further, when the communication unit 212 receives a result ofrecognition of facial behaviors of users from a captured image capturedin the other display system 400 or a result of recognition of facialbehaviors of users from a captured image captured in the other displaysystem 600, the display image generation unit 206 may receive the resultof recognition and generate a display image based on the result ofrecognition received from the communication unit 212 instead of a resultof recognition by the face recognition unit 204.

Further, when the display image generation unit 206 receives a result ofrecognition of a speaker among the users of the other informationprocessing device 500, 700 from the speech recognition unit 216, thedisplay image generation unit 206 generates a display image in which auser image representing the speaker and the presence of a speech aredisplayed in association with each other in the above-described displayimage based further on the received result of recognition of the speakeramong the users of the other information processing device 500, 700. Forexample, the display image generation unit 206 generates a display imagein which a user image representing the speaker is displayed in amagnified form.

Further, when the display image generation unit 206 further receives aresult of recognition of a speaker among the users of the informationprocessing device 200 from the speech recognition unit 216, the displayimage generation unit 206 generates a display image in which a userimage representing the speaker and the presence of a speech aredisplayed in association with each other based further on the receivedresult of recognition of the speaker among the users of the informationprocessing device 200.

Further, when the communication unit 212 receives a result ofrecognition of a speaker from a speech input to the other informationprocessing device 500 or a result of recognition of a speaker from aspeech input to the other information processing device 700, the displayimage generation unit 206 may receive the result of recognition andgenerate a display image based on the result of recognition receivedfrom the communication unit 212 instead of a result of recognition bythe speech recognition unit 216.

Further, when the display image generation unit 206 further receives aresult of recognition of contents of a speech by a speaker among theusers of the other information processing device 500, 700 from thespeech recognition unit 216, the display image generation unit 206generates a display image in which a user image representing thespeaker, the presence of a speech, and the contents of the speech aredisplayed in association with one another based further on the receivedresult of recognition of the contents of the speech by the speaker amongthe users of the other information processing device 500, 700. Forexample, the display image generation unit 206 generates a display imagein which a character image representing the contents of the speech isdisplayed within a dialogue balloon from a user image representing thespeaker.

Further, when the display image generation unit 206 further receives aresult of recognition of contents of a speech by a speaker among theusers of the information processing device 200 from the speechrecognition unit 216, the display image generation unit 206 generates adisplay image in which a user image representing the speaker, thepresence of a speech, and the contents of the speech are displayed inassociation with one another based further on the received result ofrecognition of the contents of the speech by the speaker among the usersof the information processing device 200.

Further, when the communication unit 212 receives a result ofrecognition of contents of a speech by a speaker from a speech input tothe other information processing device 500 or a result of recognitionof contents of a speech by a speaker from a speech input to the otherinformation processing device 700, the display image generation unit 206may receive the result of recognition and generate a display image basedon the result of recognition received from the communication unit 212instead of a result of recognition by the speech recognition unit 216.

Further, when the display image generation unit 206 receives a result ofrecognition of actions of the users of the other information processingdevice 500, 700 from the action recognition unit 220, the display imagegeneration unit 206 generates a display image in which a user imagerepresenting an operator user and a specific operation are displayed inassociation with each other based further on the received result ofrecognition of actions of the users of the other information processingdevice 500, 700. For example, the display image generation unit 206generates a display image in which a content image which the operatoruser intends to display is displayed within a dialogue balloon from auser image representing the operator user.

Further, when the display image generation unit 206 receives a result ofrecognition of actions of the users of the information processing device200 from the action recognition unit 220, the display image generationunit 206 generates a display image in which a user image representing anoperator user and a specific operation are displayed in association witheach other based further on the received result of recognition ofactions of the users of the information processing device 200.

Further, when the communication unit 212 receives a result ofrecognition of actions of users from an operation signal input to theother information processing device 500 or a result of recognition ofactions of users from an operation signal input to the other informationprocessing device 700, the display image generation unit 206 may receivethe result of recognition and generate a display image based on theresult of recognition received from the communication unit 212 insteadof a result of recognition by the action recognition unit 220.

The output unit 208 receives a signal for driving the display device 102from the display control unit 210, and transmits the received signal tothe display device 102. Further, when the output unit 208 receives aspeech input to the other information processing device 500, 700 whichis received by the communication unit 212, the output unit 208 maytransmit the received speech to a speaker unit (not shown) or the likeof the display system 100.

The display control unit 210, which is an example of a control in thedisclosure, receives a display image from the display image generationunit 206. Further, a content of a still image or a moving image issupplied to the display control unit 210. Then, the display control unit210 transmits a signal for displaying the display image generated by thedisplay image generation unit 206 on the display screen of the displaydevice 102 to the output unit 208. Further, the display control unit 210may transmit a signal for displaying the content on the display screenof the display device 102 to the output unit 208.

The communication unit 212 receives the captured image captured in theother display system 400, and the speech and the operation signal inputto the other information processing device 500 from the communicationserver 300 through the communication network 800. Further, thecommunication unit 212 receives the captured image captured in the otherdisplay system 600 and the speech and the operation signal input to theother information processing device 700 from the communication server300 through the communication network 800. Note that the communicationunit 212 may receive the captured image captured in the other displaysystem 400 and the speech and the operation signal input to the otherinformation processing device 500 directly from other informationprocessing device 500 through the communication network 800. Likewise,the communication unit 212 may receive the captured image captured inthe other display system 600 and the speech and the operation signalinput to the other information processing device 700 directly from otherinformation processing device 700 through the communication network 800.

Further, the communication unit 212 may receive a result of recognitionof users from the captured image captured in the other display system400, a result of recognition from the speech or a result of recognitionfrom the operation signal input to the other information processingdevice 500, a result of recognition of users from the captured imagecaptured in the other display system 600, or a result of recognitionfrom the speech or a result of recognition from the operation signalinput to the other information processing device 700 from thecommunication server 300 through the communication network 800. Notethat the communication unit 212 may receive a result of recognition ofusers from the captured image captured in the other display system 400and a result of recognition from the speech or a result of recognitionfrom the operation signal input to the other information processingdevice 500 directly from the other information processing device 500through the communication network 800. Likewise, the communication unit212 may receive a result of recognition of users from the captured imagecaptured in the other display system 600 and a result of recognitionfrom the speech or a result of recognition from the operation signalinput to the other information processing device 700 directly from theother information processing device 700 through the communicationnetwork 800.

Further, the communication unit 212 may receive the captured image fromthe image input unit 202, the speech from the speech input unit 214, andthe operation signal from the operation signal input unit 218, andtransmit the received captured image, speech and operation signal to thecommunication server 300 through the communication network 800. Further,the communication unit 212 may receive a result of recognition of usersfrom the captured image captured in the display system 100 from the facerecognition unit 204, a result of recognition from the speech input tothe information processing device 200 from the speech recognition unit216, and a result of recognition from the operation signal input to theinformation processing device 200 from the action recognition unit 220,and transmit the received results of recognition to the communicationserver 300 through the communication network 800. Note that thecommunication unit 212 may transmit the received captured image, speechand operation signal, the received result of recognition of users fromthe captured image, result of recognition from the speech and result ofrecognition from the operation signal directly to the other informationprocessing device 500, 700 through the communication network 800.

The speech input unit 214, which is an example of an input unit in thedisclosure, receives a speech from the users A and B, which are users ofthe information processing device 200, for example. Then, the speechinput unit 214 transmits the received (input) speech to thecommunication unit 212. Further, the speech input unit 214 may transmitthe received speech to the speech recognition unit 216.

The speech recognition unit 216, which is an example of a recognitionunit in the disclosure, receives a speech from the other informationprocessing device 500, 700 which is received by the communication unit212, and recognizes a speaker among the users of the other informationprocessing device 500, 700 from the received speech. The act of speakingin an example of the behavior in the disclosure. The speech from theother information processing device 500, 700 is an example of apredetermined signal related to the behavior of users of thecommunication target device according to the disclosure. For example,the speech recognition unit 216 recognizes a speaker by extracting afeature from the received speech, generating information about theuser's speech (hereinafter referred to as “user speech information”),and comparing the generated user speech information with user speechinformation 224 for each user which is previously generated and storedin the storage unit 222. Then, the speech recognition unit 216 transmitsa result of recognition of a speaker among the users of the informationprocessing device 500, 700 to the display image generation unit 206.Further, the speech recognition unit 216 may receive a speech from thespeech input unit 214 and recognize a speaker among the users of theinformation processing device 200 from the received speech. The speechfrom the speech input unit 214 is an example of a predetermined signalrelated to the behavior of users of the information processing deviceaccording to the disclosure. The speech recognition unit 216 may thentransmit a result of recognition of a speaker among the users of theinformation processing device 200 to the display image generation unit206 or the communication unit 212.

Further, the speech recognition unit 216 may recognize contents of aspeech by a speaker among the users of the other information processingdevice 500, 700 from the speech from the other information processingdevice 500, 700. For example, the speech recognition unit 216 analyzesthe received speech and thereby extracts and recognizes the contents ofthe speech by the user as character data. Then, the speech recognitionunit 216 may transmit a result of recognition of the contents of thespeech by the speaker among the users of the other informationprocessing device 500, 700 to the display image generation unit 206.Further, the display image generation unit 206 may recognize contents ofa speech by a speaker among the users of the information processingdevice 200 from the speech from the speech input unit 214. Then, thespeech recognition unit 216 may transmit a result of recognition of thecontents of the speech by the speaker among the users of the informationprocessing device 200 to the display image generation unit 206 or thecommunication unit 212.

The operation signal input unit 218, which is an example of an inputunit in the disclosure, receives an operation signal by the user A or B,the user of the information processing device 200, for example, which istransmitted using a remote controller (not shown), for example. Then,the operation signal input unit 218 transmits the received (input)operation signal to the communication unit 212. Further, the operationsignal input unit 218 may transmit the received operation signal to theaction recognition unit 220.

The action recognition unit 220, which is an example of a recognitionunit in the disclosure, receives an operation signal from the otherinformation processing device 500, 700 which is received by thecommunication unit 212, and recognizes actions of the users of the otherinformation processing device 500, 700 from the received operationsignal. The act of taking action is an example of the behavior in thedisclosure. The operation signal from the other information processingdevice 500, 700 is an example of a predetermined signal related to thebehavior of users of the communication target device according to thedisclosure. For example, the action recognition unit 220 recognizes auser as an operator by extracting information about equipment of anoperator user (hereinafter referred to as “user equipment information”)that is contained in the operation signal from the received operationsignal, and comparing the extracted user equipment information with userequipment information 226 for each user which is previously generatedand stored in the storage unit 222. Further, the action recognition unit220 recognizes a specific operation of the operator by extractinginformation about a specific operation which is contained in theoperation signal from the received operation signal, for example. Forexample, the action recognition unit 220 recognizes a content which theoperator user intends to display on the display screen by comparinginformation about a content which the operator user intends to displayon the display screen (hereinafter referred to as “user contentinformation”) that is contained in the information about a specificoperation with user content information 225 that is previously stored inthe storage unit 222. Then, the action recognition unit 220 transmits aresult of recognition of actions of the users of the other informationprocessing device 500, 700 to the display image generation unit 206.Further, the action recognition unit 220 may receive an operation signalfrom the operation signal input unit 218 and recognize actions of theusers of the information processing device 200 from the receivedoperation signal. The operation signal from the operation signal inputunit 218 is an example of a predetermined signal related to the behaviorof users of the information processing device according to thedisclosure. Then, the action recognition unit 220 may transmit a resultof recognition of actions of the users of the information processingdevice 200 to the display image generation unit 206 or the communicationunit 212.

The storage unit 222 stores user face information 223 that isinformation about the faces of the users of the other informationprocessing device 500, 700 or the information processing device 200,user speech information 224 that is information about the speeches ofthe users of the other information processing device 500, 700 or theinformation processing device 200, user content information 225 that isinformation about the contents of the users of the other informationprocessing device 500, 700 or the information processing device 200, anduser equipment information 226 that is information about the equipmentsused by the users of the other information processing device 500, 700 orthe information processing device 200.

It should be noted that the predetermined signal related to the behaviorof users of the communication target device according to the disclosureis not limited to the captured image, the speech and the operationsignal from the other information processing device 500, 700 describedabove, and it may be any signal from the other information processingdevice 500, 700 which is generated as a result that a user of the otherinformation processing device 500, 700 performs some acts. Further, thepredetermined signal related to the behavior of users of the informationprocessing device according to the disclosure is not limited to thecaptured image, the speech and the operation signal which are input tothe information processing device 200 described above, and it may be anysignal which is generated as a result that a user of the informationprocessing device 200 performs some acts.

3. Hardware Configuration of Information Processing Device

A hardware configuration of the information processing device 200 inFIG. 2 is described hereinbelow. FIG. 3 is an explanatory viewillustrating an example of a hardware configuration of the informationprocessing device 200 in FIG. 2.

In FIG. 3, the information processing device 200 includes an MPU 230, aROM 232, a RAM 234, a recording medium 236, an input/output interface238, an operation input device 240, a display device 242, and acommunication interface 244, for example. Further, in the informationprocessing device 200, the components are connected through a bus 246 asa data transmission line, for example.

The MPU 230 is configured as an MPU (Micro Processing Unit), anintegrated circuit in which a plurality of circuits for implementingvarious functions such as image processing are integrated or the like,and functions as a control unit (not shown) that controls theinformation processing device 200 as a whole. Further, the MPU 230 actsas the face recognition unit 204, the display image generation unit 206,the display control unit 210, the speech recognition unit 216 and theaction recognition unit 220 in the information processing device 200.

The ROM 232 stores programs to be used by the MPU 230, control data suchas operation parameters and the like. The RAM 234 temporarily stores aprogram executed by the MPU 230, for example.

The recording medium 236 functions as the storage unit 222, and storesvarious kinds of data (or database) such as the user face information,the user speech information, the user content information and the userequipment information, applications, and the like. The recording medium236 may be a magnetic recording medium such as hard disk, nonvolatilememory such as EEPROM (Electrically Erasable and Programmable Read OnlyMemory), flash memory, MRAM (Magnetoresistive Random Access Memory),FeRAM (Ferroelectric Random Access Memory), PRAM Phase change RandomAccess Memory), and the like. Further, the information processing device200 may include a recording medium 236 which is removable from theinformation processing device 200.

The input/output interface 238 connects to the operation input device240 and the display device 242, for example. Further, the input/outputinterface 238 may act as the image input unit 202, the output unit 208,the speech input unit 214 and the operation signal input unit 218. Theoperation input device 240 functions as an operation unit (not shown),and the display device 242 functions as a display unit 254 in FIG. 15,which is described later. The input/output interface 238 may be a USB(Universal Serial Bus) terminal, a DVI (Digital Visual Interface)terminal, an HDMI (High-Definition Multimedia Interface) terminal, orvarious processing circuits, for example. Further, the operation inputdevice 240 is mounted on the information processing device 200 andconnected with the input/output interface 238 inside the informationprocessing device 200, for example. The operation input device 240 maybe a button, a cursor key, a rotating selector such as Jog Dial, or acombination of those, for example. Furthermore, the display device 242is mounted on the information processing device 200 and connected withthe input/output interface 238 inside the information processing device200, for example. The display device 242 may be a liquid crystal display(LCD), or an organic EL display (organic ElectroLuminescence display.Also called an OLED display (Organic Light Emitting Diode display)) andthe like, for example. Note that the input/output interface 238 can alsoconnect to external devices of the information processing device 200,such as an operation input device (e.g. a keyboard, a mouse), a displaydevice (e.g. an external display device such as the display device 102)and an imaging device (e.g. the imaging device 104). Further, thedisplay device 242 may be a device capable of display and useroperation, such as a touch screen, for example.

The communication interface 244 is a communication means included in theinformation processing device 200, and functions as the communicationunit 212 for performing wireless/wired communication with externaldevices such as the communication server 300 and the other informationprocessing device 500, 700, for example, through the communicationnetwork 800 (or directly). The communication interface 244 may be acommunication antenna and an RF circuit (wireless communication),IEEE802.15.1 port and a transmitting and receiving circuit (wirelesscommunication), IEEE802.11b port and a transmitting and receivingcircuit (wireless communication), or a LAN terminal and a transmittingand receiving circuit (wired communication) and the like, for example.

It should be noted that the hardware configuration of the informationprocessing device 200 according to the embodiment is not limited to theconfiguration shown in FIG. 3. For example, the information processingdevice 200 may include a speech output device composed of a DSP (DigitalSignal Processor), an amplifier, a speaker and the like, which acts as aspeech output unit (not shown).

Further, the information processing device 200 may include an imagingdevice composed of a lens/image pickup device and a signal processingcircuit, which acts as an imaging unit 252 in FIG. 15, for example. Inthis case, the information processing device 200 can process thecaptured image generated in its own device. The lens/image pickup deviceis made up of an optical lens and an image sensor composed of aplurality of image pickup devices such as CCD (Charge Coupled Device) orCMOS (Complementary Metal Oxide Semiconductor), for example. The signalprocessing circuit includes an AGC (Automatic Gain Control) circuit andan ADC (Analog to Digital Converter), for example, and converts ananalog signal generated by the image pickup device into a digital signal(image data) and performs various signal processing. The signalprocessing that is performed by the signal processing circuit may beWhite Balance correction, interpolation, color tone correction, gammacorrection, YCbCr conversion, edge enhancement, coding and the like, forexample.

Further, the information processing device 200 may have a configurationin which the operation input device 240 and the display device 242 shownin FIG. 3 are not included, for example.

4. Display Process

A display process that is executed by the information processing device200 in FIG. 2 is described hereinbelow. FIG. 4 is a flowchart of thedisplay process executed by the information processing device 200 inFIG. 2.

Referring to FIG. 4, when an instruction for network connection with theusers C and D and the users E and F is given from the user A, forexample, in the information processing device 200, the communicationunit 212 becomes able to communicate with the communication server 300through the communication network 800 (Step S100). Note that, in StepS100, the communication unit 212 may become able to communicate with theother information processing device 500, 700 directly through thecommunication network 800.

Next, the communication unit 212 transmits a captured image which isgenerated by imaging in the imaging device 104 in the display system 100and received via the image input unit 202 to the communication server300 through the communication network 800 (Step S102). Note that, inStep S102, the communication unit 212 may transmit a result ofrecognition of the users of the information processing device 200 and aresult of recognition of facial behaviors of the users of theinformation processing device 200 which are received from the facerecognition unit 204 to the communication server 300 through thecommunication network 800. Further, in Step S102, the communication unit212 may transmit the captured image, a result of recognition of theusers, and a result of recognition of facial behaviors of the users tothe other information processing device 500, 700 directly through thecommunication network 800.

Then, the communication unit 212 transmits a speech which has been inputto the information processing device 200 and received via the speechinput unit 214 to the communication server 300 through the communicationnetwork 800 (Step S104). Note that, in Step S104, the communication unit212 may transmit a result of recognition of a speaker among the users ofthe information processing device 200 and a result of recognition ofcontents of a speech by a speaker among the users of the informationprocessing device 200 which are received from the speech recognitionunit 216 to the communication server 300 through the communicationnetwork 800. Further, in Step S104, the communication unit 212 maytransmit the speech, a result of recognition of a speaker among theusers, a result of recognition of contents of a speech by a speakeramong the users to the other information processing device 500, 700directly through the communication network 800.

Then, the communication unit 212 transmits an operation signal which hasbeen input to the information processing device 200 and received via theoperation signal input unit 218 to the communication server 300 throughthe communication network 800 (Step S106). Note that, in Step S106, thecommunication unit 212 may transmit a result of recognition of actionsof the users of the information processing device 200 which is receivedfrom the action recognition unit 220 to the communication server 300through the communication network 800. Further, in Step S106, thecommunication unit 212 may transmit the operation signal and a result ofrecognition of actions of the users to the other information processingdevice 500, 700 directly through the communication network 800.

After that, the communication unit 212 receives a captured image whichhas been captured in the display system 400 and transmitted from theother information processing device 500 from the communication server300. Further, the communication unit 212 receives a captured image whichhas been captured in the display system 600 and transmitted from theother information processing device 700 from the communication server300 (Step S108). Note that, in Step S108, the communication unit 212 mayreceive a result of recognition of users and a result of recognition offacial behaviors of users from the captured image which has beencaptured in the display system 400 and transmitted from the otherinformation processing device 500, and a result of recognition of usersand a result of recognition of facial behaviors of users from thecaptured image which has been captured in the display system 600 andtransmitted from the other information processing device 700 from thecommunication server 300. Further, in Step S108, the communication unit212 may receive the captured image, a result of recognition of users,and a result of recognition of facial behaviors of the users from theother information processing device 500, 700 directly through thecommunication network 800.

Then, the communication unit 212 receives a speech which has been inputto the other information processing device 500 and transmitted from theother information processing device 500 from the communication server300. Further, the communication unit 212 receives a speech which hasbeen input to the other information processing device 700 andtransmitted from the other information processing device 700 from thecommunication server 300 (Step S110). Note that, in Step S110, thecommunication unit 212 may receive a result of recognition of a speakeramong users and a result of recognition of contents of a speech by aspeaker among users from the speech which has been input to the otherinformation processing device 500 and transmitted from the otherinformation processing device 500, and a result of recognition of aspeaker among users and a result of recognition of contents of a speechby a speaker among users from the speech which has been input to theother information processing device 700 and transmitted from the otherinformation processing device 700 from the communication server 300.Further, in Step S110, the communication unit 212 may receive thespeech, a result of recognition of a speaker among the users, a resultof recognition of contents of a speech by a speaker among the users fromthe other information processing device 500, 700 directly through thecommunication network 800.

Then, the communication unit 212 receives an operation signal which hasbeen input to the other information processing device 500 andtransmitted from the other information processing device 500 from thecommunication server 300. Further, the communication unit 212 receivesan operation signal which has been input to the other informationprocessing device 700 and transmitted from the other informationprocessing device 700 from the communication server 300 (Step S112).Note that, in Step S112, the communication unit 212 may receive a resultof recognition of actions of users from the operation signal which hasbeen input to the other information processing device 500 andtransmitted from the other information processing device 500, and aresult of recognition of actions of users from the operation signalwhich has been input to the other information processing device 700 andtransmitted from the other information processing device 700 from thecommunication server 300. Further, in Step S112, the communication unit212 may receive the operation signal and a result of recognition ofactions of the users from the other information processing device 500,700 directly through the communication network 800.

After that, the information processing device 200 performs user displayprocessing in FIG. 5, which is described later (Step S114) so that userimages representing the users of the information processing device 200and the other information processing device 500, 700 are displayed foreach user, and a user image representing a user who has performed abehavior and a detail of the behavior are displayed in association witheach other on the display screen of the display device 102, for example.

FIG. 5 is a flowchart of the user display processing that is performedin Step S114 in the display process of FIG. 4.

Referring to FIG. 5, the face recognition unit 204 first receives acaptured image from the other information processing device 500, 700which is received by the communication unit 212, detects faces of theusers of the information processing device 500, 700 from the receivedcaptured image, and recognizes users of the information processingdevice 500, 700, i.e., the users C and D or the users E and F, based onthe detection result of faces (Step S200). Further, in Step S200, theface recognition unit 204 may detect faces of the users of theinformation processing device 500, 700 from the received captured image,and recognize facial behavior of the users of the information processingdevice 500, 700, i.e., the users C and D or the users E and F, based onthe detection result of faces. Furthermore, in Step S200, the facerecognition unit 204 may receive a captured image from the image inputunit 202, detect faces of the users of the information processing device200 from the received captured image, and recognize users of theinformation processing device 200, i.e., the users A and B or recognizefacial behavior of the users of the information processing device 200,i.e., the users A and B, based on the detection result of faces.

Next, the speech recognition unit 216 receives a speech from the otherinformation processing device 500, 700 which is received by thecommunication unit 212, and recognizes a speaker among the users of theinformation processing device 500, 700 from the received speech (StepS202). Further, in Step S202, the speech recognition unit 216 mayreceive a speech from the other information processing device 500, 700which is received by the communication unit 212, and recognizes contentsof a speech by a speaker among the users of the information processingdevice 500, 700 from the received speech. Further, in Step S202, thespeech recognition unit 216 may receive a speech from the speech inputunit 214 and recognize a speaker among the users of the informationprocessing device 200 or recognize contents of a speech by a speakeramong the users from the received speech.

Then, the action recognition unit 220 receives an operation signal fromthe other information processing device 500, 700 which is received bythe communication unit 212, and recognizes actions of the users of theinformation processing device 500, 700 from the received operationsignal (Step S204). Further, in Step S204, the action recognition unit220 may receive an operation signal from the operation signal input unit218 and recognize actions of the users of the information processingdevice 200 from the received operation signal.

After that, the display image generation unit 206 receives a result ofrecognition of the users of the other information processing device 500,700 from the face recognition unit 204, and receives the captured imagecaptured in the other display system 400 or the captured image capturedin the other display system 600 which is received by the communicationunit 212. Then, based on the received result of recognition of the usersof the other information processing device 500, 700, the display imagegeneration unit 206 generates a display image in which user imagesrepresenting the users of the other information processing device 500,700 are displayed for each user (Step S206).

Further, in Step S206, when the display image generation unit 206receives a captured image captured in the display system 100 from theimage input unit 202, receives a result of recognition of the users ofthe information processing device 200 and a result of recognition of theusers of the other information processing device 500, 700 from the facerecognition unit 204, and receives a captured image captured in theother display system 400 or a captured image captured in the otherdisplay system 600 which is received by the communication unit 212, thedisplay image generation unit 206 generates a display image in whichuser images representing the users of the information processing device200 and the other information processing device 500, 700 are displayedfor each user based on the received results of recognition of the usersof the information processing device 200 and the users of the otherinformation processing device 500, 700.

Further, in Step S206, when the communication unit 212 receives a resultof recognition of users from a captured image captured in the otherdisplay system 400 or a result of recognition of users from a capturedimage captured in the other display system 600, the display imagegeneration unit 206 may receive the result of recognition and generate adisplay image based on the result of recognition received from thecommunication unit 212 instead of a result of recognition by the facerecognition unit 204.

Further, in Step S206, when the display image generation unit 206further receives a result of recognition of facial behaviors of theusers of the other information processing device 500, 700 from the facerecognition unit 204, the display image generation unit 206 generates adisplay image in which a user image representing a user whose facialbehavior has changed and the facial behavior are displayed inassociation with each other in the above-described display image basedfurther on the received result of recognition of facial behaviors of theusers of the other information processing device 500, 700.

Further, in Step S206, when the display image generation unit 206further receives a result of recognition of facial behaviors of theusers of the information processing device 200 from the face recognitionunit 204, the display image generation unit 206 generates a displayimage in which a user image representing a user whose facial behaviorhas changed and the facial behavior are displayed in association witheach other based further on the received result of recognition of facialbehaviors of the users of the information processing device 200.

Further, in Step S206, when the communication unit 212 receives a resultof recognition of facial behaviors of users from a captured imagecaptured in the other display system 400 or a result of recognition offacial behaviors of users from a captured image captured in the otherdisplay system 600, the display image generation unit 206 may receivethe result of recognition and generate a display image based on theresult of recognition received from the communication unit 212 insteadof a result of recognition by the face recognition unit 204.

Further, in Step S206, when the display image generation unit 206receives a result of recognition of a speaker among the users of theother information processing device 500, 700 from the speech recognitionunit 216, the display image generation unit 206 generates a displayimage in which a user image representing the speaker and the presence ofa speech are displayed in association with each other in theabove-described display image based further on the received result ofrecognition of the speaker among the users of the other informationprocessing device 500, 700.

Further, in Step S206, when the display image generation unit 206further receives a result of recognition of a speaker among the users ofthe information processing device 200 from the speech recognition unit216, the display image generation unit 206 generates a display image inwhich a user image representing the speaker and the presence of a speechare displayed in association with each other based further on thereceived result of recognition of the speaker among the users of theinformation processing device 200.

Further, in Step S206, when the communication unit 212 receives a resultof recognition of a speaker from a speech input to the other informationprocessing device 500 or a result of recognition of a speaker from aspeech input to the other information processing device 700, the displayimage generation unit 206 may receive the result of recognition andgenerate a display image based on the result of recognition receivedfrom the communication unit 212 instead of a result of recognition bythe speech recognition unit 216.

Further, in Step S206, when the display image generation unit 206further receives a result of recognition of contents of a speech by aspeaker among the users of the other information processing device 500,700 from the speech recognition unit 216, the display image generationunit 206 generates a display image in which a user image representingthe speaker, the presence of a speech, and the contents of the speechare displayed in association with one another based further on thereceived result of recognition of the contents of the speech by thespeaker among the users of the other information processing device 500,700.

Further, in Step S206, when the display image generation unit 206further receives a result of recognition of contents of a speech by aspeaker among the users of the information processing device 200 fromthe speech recognition unit 216, the display image generation unit 206generates a display image in which a user image representing thespeaker, the presence of a speech, and the contents of the speech aredisplayed in association with one another based further on the receivedresult of recognition of the contents of the speech by the speaker amongthe users of the information processing device 200.

Further, in Step S206, when the communication unit 212 receives a resultof recognition of contents of a speech by a speaker from a speech inputto the other information processing device 500 or a result ofrecognition of contents of a speech by a speaker from a speech input tothe other information processing device 700, the display imagegeneration unit 206 may receive the result of recognition and generate adisplay image based on the result of recognition received from thecommunication unit 212 instead of a result of recognition by the speechrecognition unit 216.

Further, in Step S206, when the display image generation unit 206receives a result of recognition of actions of the users of the otherinformation processing device 500, 700 from the action recognition unit220, the display image generation unit 206 generates a display image inwhich a user image representing an operator user and a specificoperation are displayed in association with each other based further onthe received result of recognition of actions of the users of the otherinformation processing device 500, 700.

Further, in Step S206, when the display image generation unit 206receives a result of recognition of actions of the users of theinformation processing device 200 from the action recognition unit 220,the display image generation unit 206 generates a display image in whicha user image representing an operator user and a specific operation aredisplayed in association with each other based further on the receivedresult of recognition of actions of the users of the informationprocessing device 200.

Further, in Step S206, when the communication unit 212 receives a resultof recognition of actions of users from an operation signal input to theother information processing device 500 or a result of recognition ofactions of users from an operation signal input to the other informationprocessing device 700, the display image generation unit 206 may receivethe result of recognition and generate a display image based on theresult of recognition received from the communication unit 212 insteadof a result of recognition by the action recognition unit 220.

After that, the display control unit 210 receives a display image fromthe display image generation unit 206, and transmits a signal fordisplaying the received display image on the display screen of thedisplay device 102 to the output unit 208. Then, the output unit 208transmits the signal for displaying the received display image to thedisplay device 102. As a result, a display image in which user imagesrepresenting the users of the other information processing device 500,700 are displayed for each user, a display image in which user imagesrepresenting the users of the information processing device 200 and theother information processing device 500, 700 are displayed for eachuser, a display image in which user images representing the users of theother information processing device 500, 700 are displayed for each userand a user image representing a user who has performed a behavior and adetail of the behavior are displayed in association with each other, anda display image in which user images representing the users of theinformation processing device 200 and the other information processingdevice 500, 700 are displayed for each user and a user imagerepresenting a user who has performed a behavior and a detail of thebehavior are displayed in association with each other are displayed onthe display screen of the display device 102 (Step S208). The processingthereby ends.

According to the display process of FIG. 4, a display image in whichuser images representing the users of the other information processingdevice 500, 700 are displayed for each user is displayed on the displayscreen of the display device 102. Thus, even when there are a pluralityof users of the other information processing device 500, for example, auser of the information processing device 200 can be surely aware of theexistence of any user of the other information processing device 500because the user images representing the respective users of the otherinformation processing device 500 are displayed for each of the users onthe display screen of the display device 102.

Further, according to the display process of FIG. 4, a display image inwhich user images representing the users of the information processingdevice 200 and the other information processing device 500, 700 aredisplayed for each user is displayed on the display screen of thedisplay device 102. Thus, a user of the information processing device200 can be surely aware of the existence of any user of the otherinformation processing device 500 and can feel a sense of realism, asense of togetherness, a sense of closeness and the like as ifcommunicating with them in the same place.

Further, according to the display process of FIG. 4, a display image inwhich user images representing the users of the other informationprocessing device 500, 700 are displayed for each user, and a user imagerepresenting a user who has performed a behavior and a detail of thebehavior are displayed in association with each other is displayed onthe display screen of the display device 102. Thus, a user of theinformation processing device 200 can be surely aware of the existenceof any user of the other information processing device 500 and, evenwhen there are a plurality of users of the other information processingdevice 500, for example, can be surely aware of which user of the otherinformation processing device 500 has performed a behavior because auser image representing a user who has performed a behavior and a detailof the behavior are displayed in association with each other on thedisplay screen of the display device 102.

Further, according to the display process of FIG. 4, a display image inwhich user images representing the users of the information processingdevice 200 and the other information processing device 500, 700 aredisplayed for each user, and a user image representing a user who hasperformed a behavior and a detail of the behavior are displayed inassociation with each other is displayed on the display screen of thedisplay device 102. Thus, a user of the information processing device200 can be surely aware of the existence of any user of the otherinformation processing device 500, can feel a sense of realism, a senseof togetherness, a sense of closeness and the like as if communicatingwith them in the same place, and further can be surely aware of whichuser of the other information processing device 500 has performed abehavior.

5. Examples

Examples of the disclosure are described hereinbelow. FIGS. 6A to 6D andFIGS. 7A to 7D are explanatory views illustrating an example 1 of thedisclosure. In the example 1, a case where an instruction for networkconnection with the users C and D is given by the user A, for example,in the information processing device 200 of FIG. 2, and a display imagein which user images representing the users of the informationprocessing device 200 and the other information processing device 500,700 are displayed for each user, and a user image representing a userwho has performed a behavior and a detail of the behavior are displayedin association with each other is displayed on the display screen of thedisplay device 102 in FIG. 2 and the display screen in the displaysystem 400 is described by way of illustration.

First, the information processing device 200 receives a captured imageshown in FIG. 6A and a captured image shown in FIG. 6B, and recognizesthe users A and B of the information processing device 200 and the usersC and D of the other information processing device 500 by theabove-described face recognition. Then, as shown in FIG. 6C, theinformation processing device 200 displays a display image in which auser image 110 representing the user A of the information processingdevice 200, a user image 112 representing the user B of the informationprocessing device 200, a user image 114 representing the user C of theother information processing device 500, and a user image 116representing the user D of the other information processing device 500are displayed for each user on the display screen of the display device102. Likewise, as shown in FIG. 6D, the other information processingdevice 500 displays a display image in which a user image 410representing the user A of the information processing device 200, a userimage 412 representing the user B of the information processing device200, a user image 414 representing the user C of the other informationprocessing device 500, and a user image 416 representing the user D ofthe other information processing device 500 are displayed for each useron the display screen of a display device 402 in the display system 400.

Next, as shown in FIG. 7A, when a speech saying “I want to see Mt.Fuji”, for example, is made as a behavior by the user D, the otherinformation processing device 500 receives the speech and recognizes theuser D as a speaker among the users of the other information processingdevice 500 and the contents of the speech by the user D by theabove-described speech recognition. Then, as shown in FIG. 7B, the otherinformation processing device 500 displays a display image in which acharacter image representing the contents of the speech, which is theimage “I want to see Mt. Fuji”, is displayed within a dialogue balloon418 from the user image 416 representing the user D as a speaker.Likewise, the information processing device 200 displays a display imagein which a character image representing the contents of the speech,which is the image “I want to see Mt. Fuji”, is displayed within adialogue balloon (not shown) from the user image 116 representing theuser D as a speaker on the display screen of the display device 102,although not shown.

Further, as shown in FIG. 7A, when transmission of the character data “Iwant to see Mt. Fuji” is performed using a remote controller, forexample, as a behavior by the user D, the other information processingdevice 500 receives the operation signal and recognizes the user D as anoperator among the users of the other information processing device 500and a specific operation performed by the operator by theabove-described action recognition. Then, as shown in FIG. 7B, the otherinformation processing device 500 displays a display image in which acharacter image representing the specific operation by the operator,which is the image “I want to see Mt. Fuji”, is displayed within adialogue balloon 418 from the user image 416 representing the user D asa speaker on the display screen of the display device 402. Likewise, theinformation processing device 200 displays a display image in which acharacter image representing the specific operation by the operator,which is the image “I want to see Mt. Fuji”, is displayed within adialogue balloon (not shown) from the user image 116 representing theoperator user D on the display screen of the display device 102,although not shown.

Then, as shown in FIG. 7C, when transmission of a request for display of“the content of Mt. Fuji” is performed using a remote controller, forexample, as a behavior by the user B, the information processing device200 receives the operation signal and recognizes the user B as anoperator among the users of the information processing device 200 and aspecific operation performed by the operator by the above-describedaction recognition. Then, as shown in FIG. 7D, the informationprocessing device 200 displays a display image in which a content imagewhich the operator user B intends to display, which is the image of “thecontent of Mt. Fuji”, is displayed within a dialogue balloon 118 fromthe user image 112 representing the operator user B on the displayscreen of the display device 102. Likewise, the other informationprocessing device 500 displays a display image in which a content imagewhich the operator user B intends to display, which is the image of “thecontent of Mt. Fuji”, is displayed within a dialogue balloon (not shown)from the user image 412 representing the operator user B on the displayscreen of the display device 402, although not shown. At this time, theother information processing device 500 may receive a content imagewhich the operator user B intends to display, which is the image of “thecontent of Mt. Fuji”, from the information processing device 200 orreceive the content image from another device through the communicationnetwork 800, for example.

According to the example 1, a user of the information processing device200 and the other information processing device 500 can be surely awareof the existence of any user of the information processing device 200and the other information processing device 500, can feel a sense ofrealism, a sense of togetherness, a sense of closeness and the like asif communicating with them in the same place, and further can be surelyaware of which user of the information processing device 200 and theother information processing device 500 has performed a behavior.

FIGS. 8A to 8D are explanatory views illustrating an example 2 of thedisclosure. In the example 2, a case where an instruction for networkconnection with the users C and D is given by the user A, for example,in the information processing device 200 of FIG. 2, and the user A goesaway during network connection with the users C and D is described byway of illustration.

When the user A has gone away during network connection with the users Cand D as shown in FIGS. 8A and 8B, a display image in which a user image120 representing the user A of the information processing device 200 isdeleted at the same time when the user A goes away is displayed on thedisplay screen of the display device 102. Likewise, a display image inwhich a user image (not shown) representing the user A of theinformation processing device 200 is deleted at the same time when theuser A goes away is displayed on the display screen of the displaydevice 402 in the display system 400, although not shown.

According to the example 2, because the user image 120 representing theuser A of the information processing device 200 is deleted at the sametime when the user A goes away in the display image which is displayedon the display screen of the display device 402, a user of the otherinformation processing device 500 can be surely aware that the user A ofthe information processing device 200 has gone away.

FIGS. 9A to 9D are explanatory views illustrating an example 3 of thedisclosure. In the example 3, a case where an instruction for networkconnection with the users C and D is given by the user A, for example,in the information processing device 200 of FIG. 2, and the facialbehavior of the user A has changed to smiling during network connectionwith the users C and D is described by way of illustration.

When the facial behavior of the user A has changed to smiling duringnetwork connection with the users C and D as shown in FIGS. 9A to 9D, adisplay image in which a user image 130 representing the user A whosefacial behavior has changed to smiling is displayed like jumping isdisplayed on the display screen of the display device 102. Likewise, adisplay image in which a user image (not shown) representing the user Awhose facial behavior has changed to smiling is displayed like jumpingis displayed on the display screen of the display device 402 in thedisplay system 400, although not shown.

According to the example 3, because the user image 130 representing theuser A whose facial behavior has changed to smiling is displayed likejumping in the display image which is displayed on the display screen ofthe display device 402, a user of the other information processingdevice 500 can be surely aware that the facial behavior of the user A ofthe information processing device 200 has changed to smiling.

FIGS. 10A and 10B are explanatory views illustrating an example 4 of thedisclosure. In the example 4, a case where an instruction for networkconnection with the users C and D is given by the user A, for example,in the information processing device 200 of FIG. 2, and a display imagein which user images representing the users of the informationprocessing device 200 and the other information processing device 500,700 are displayed for each user, and a user image representing a userwho has performed a behavior and a detail of the behavior are displayedin association with each other is displayed on the display screen of thedisplay device 102 in FIG. 2 and the display screen in the displaysystem 400 is described by way of illustration.

As shown in FIGS. 10A and 10B, the information processing device 200recognizes the users A and B of the information processing device 200and the users C and D of the other information processing device 500 bythe above-described face recognition and further recognizes the genderof the users A and B of the information processing device 200 and theusers C and D of the other information processing device 500, and thendisplays a display image in which a user image 140 representing the userA of the information processing device 200, a user image 144representing the user B of the information processing device 200, a userimage 142 representing the user C of the other information processingdevice 500, and a user image 146 representing the user D of the otherinformation processing device 500 are displayed for each user and foreach gender on the display screen of the display device 102. Likewise,though not shown, the other information processing device 500 recognizesthe users A and B of the information processing device 200 and the usersC and D of the other information processing device 500 by theabove-described face recognition and further recognizes the gender ofthe users A and B of the information processing device 200 and the usersC and D of the other information processing device 500, and thendisplays a display image in which a user image (not shown) representingthe user A of the information processing device 200, a user image (notshown) representing the user B of the information processing device 200,a user image (not shown) representing the user C of the otherinformation processing device 500, and a user image (not shown)representing the user D of the other information processing device 500are displayed for each user and for each gender on the display screen ofthe display device 402.

According to the example 4, a user of the information processing device200 and the other information processing device 500 can be surely awareof the gender of any user of the information processing device 200 andthe other information processing device 500.

FIGS. 11A to 11E are explanatory views illustrating an example 5 of thedisclosure. In the example 5, a case where an instruction for networkconnection with the users A and B is given by the user C, for example,in the other information processing device 500 of FIG. 2, and a contentimage 430 is newly displayed on the display screen in the display system400 in FIG. 2 when a display image 420 which is generated by a displayimage generation unit (not shown) of the other information processingdevice 500 is being displayed on the display screen in the displaysystem 400 is described by way of illustration.

As shown in FIGS. 11A to 11E, in the case where the content image 430 isdisplayed on the display screen in the display system 400 when thedisplay image 420 is being displayed thereon, the display image 420 isgradually shifted to the right to fade out of the screen, and thecontent image 430 is shifted from the left to the right to appear in thescreen, thereby displaying the content image 430. Then, as shown in FIG.11E, a display 440 indicating that it is under network connection isdisplayed on the display screen in the display system 400.

According to the example 5, a user of the other information processingdevice 500 can be surely aware that it is under network connection withthe users A and B.

FIGS. 12A and 12B are explanatory views illustrating an example 6 of thedisclosure. In the example 6, a case where an instruction for networkconnection with the user C is given by the user A, for example, in theinformation processing device 200 of FIG. 2, and a content image and adisplay image generated by the display image generation unit 206 aredisplayed in superposition on the display screen of the display device102 is described by way of illustration.

As shown in FIGS. 12A and 12B, in the case where a content image and adisplay image generated by the display image generation unit 206 aredisplayed in superposition on the display screen of the display device102, user images 150 and 152 are gradually moved offscreen. Then, asshown in FIG. 12B, when a user represented by the user image 150, 152performs some act of behavior such as speech, wave-like behavior, forexample, is made from the position corresponding to where the user image150, 152 has been moved.

According to the example 6, a user of the information processing device200 can view the entire content image and can be surely aware that it isunder network connection with the user C.

FIGS. 13A to 13L are explanatory views illustrating an example 7 of thedisclosure. In the example 7, a case where an instruction for networkconnection with the users A and B and the users C and D is given by theuser E, for example, in the information processing device 700 of FIG. 2,and a display image in which user images representing the users of theinformation processing device 200 and the other information processingdevice 500, 700 are displayed for each user, and a user imagerepresenting a user who has performed a behavior and a detail of thebehavior are displayed in association with each other is displayed onthe display screen in the display system 600 of FIG. 2 is described byway of illustration.

As shown in FIGS. 13A to 13L, on the display screen in the displaysystem 600, a common content area A1 as an area where the users of theinformation processing device 200 and the other information processingdevice 500, 700 share contents, and a user-specific content area A2 asan area which represents contents of the users E and F of the otherinformation processing device 700 are set. Then, when a content image602 is selected as a content to be displayed in the common content areaA1 by the user E, for example, a content image 604 corresponding to thecontent image 602 is displayed and gradually scaled up in a lower partas a position corresponding to the user E in the common content area A1(FIGS. 13B to 13D). Further, when a content image (not shown) isselected as a content to be displayed in the common content area A1 bythe user A, for example, a user image representing the user A is scaledup, and a content image 606 corresponding to the content image (notshown) which is selected by the user A is displayed and gradually scaledup in a lower right part as a position corresponding to the user A inthe common content area A1 (FIGS. 13F to 13H). Further, when a contentimage (not shown) is selected as a content to be displayed in the commoncontent area A1 by the user C, for example, a user image representingthe user C is scaled up, and a content image 608 corresponding to thecontent image (not shown) which is selected by the user C is displayedand gradually scaled up in an upper left part as a positioncorresponding to the user C in the common content area A1 (FIGS. 13J to13L). At this time, the content image 606 selected by the user A mayfade out as shown in FIG. 13K.

According to the example 7, a user of the other information processingdevice 700 can be surely aware of the existence of any user of theinformation processing device 200 and the other information processingdevice 500, can feel a sense of realism, a sense of togetherness, asense of closeness and the like as if communicating with them in thesame place, and further can be surely aware of which user of theinformation processing device 200 and the other information processingdevice 500 has performed a behavior.

6. Functional Configuration of Information Processing Device SecondEmbodiment

A functional configuration of an information processing device accordingto a second embodiment of the disclosure is described. FIG. 14 is anexplanatory view illustrating an example of a functional configurationof an information processing device according to the embodiment. Aninformation processing device 250 as the information processing deviceaccording to the embodiment is different from the information processingdevice according to the first embodiment described above only in that itincludes an imaging unit 252 and a display unit 254, and thereforeexplanation of redundant elements and functions is omitted, andexplanation of different elements and functions is provided below.

Note that FIG. 14 shows users A and B, which are users of theinformation processing device 250, a communication network 800 to whichthe information processing device 250 can connect, a communicationserver 300 and other information processing devices 550 and 750 that canconnect to the communication network 800, users C and D, which are usersof the other information processing device 550, and users E and F, whichare users of the other information processing device 750. The otherinformation processing devices 550 and 750 have the same configurationas the information processing device 250, and detailed explanationthereof is omitted. The other information processing devices 550 and 750are examples of a communication target device in the disclosure.

In FIG. 13, the information processing device 250 includes an imagingunit 252, a face recognition unit 204, a display image generation unit206, a display unit 254, a display control unit 210, a communicationunit 212, a speech input unit 214, a speech recognition unit 216, anoperation signal input unit 218, an action recognition unit 220, and astorage unit 222, for example.

The imaging unit 252 can capture still images or moving images of theusers A and B looking at the display screen of the display unit 254.Then, the imaging unit 252 can transmit a captured image generated byimaging to the communication unit 212, the face recognition unit 204 andthe display image generation unit 206.

In the display unit 254, a display image in which user imagesrepresenting the users of the information processing device 250 and theother information processing device 550, 750 are displayed for eachuser, and a user image representing a user who has performed a behaviorand a detail of the behavior are displayed in association with eachother can be displayed, for example.

According to the embodiment, the information processing device 250executes the display process of FIG. 4 which is described earlier,thereby having the same effect as described earlier in the firstembodiment. Further, according to the embodiment, the need forseparately preparing the display system 100 shown in FIG. 2 can beeliminated, thereby allowing easier implementation.

Further, an object of the disclosure may be accomplished also byproviding a storage medium, such as a non-transitory storage medium,storing program code of software that implements the functions of eachof the embodiments described above to a system or a device, and readingand executing the program code stored in the storage medium by acomputer (or CPU, MPU etc.) of the system or the device.

In this case, the program code which is read from the storage mediumimplements the functions of each of the embodiments described above, andthe program code and the storage medium storing the program codeconstitute the disclosure.

Further, the storage medium for providing the program code may be floppy(registered trademark) disk, hard disk, magneto-optical disk, opticaldisk such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW or DVD+RW,magnetic tape, nonvolatile memory card, ROM or the like, for example.Alternatively, the program code may be downloaded over a network.

Further, the disclosure is not limited to the case where a computerexecutes the read program code to thereby implement the functions ofeach of the embodiments described above, and the case where an OS(Operating System) or the like running on a computer performs a part orthe whole of actual processing based on instructions of the program codeto thereby implement the functions of each of the embodiments describedabove is also included.

Further, the case where the program code read from the storage medium iswritten into a memory in a function expansion board inserted to acomputer or a function expansion unit connected to a computer, and thena CPU or the like in the expansion board or the expansion unit performsa part or the whole of actual processing for the expanded functionsbased on instructions of the program code to thereby implement thefunctions of each of the embodiments described above is also included.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

For example, the communication server 300 may recognize users or facialbehavior of users of the other information processing device 500, 700from the captured image from the other information processing device500, 700. Further, the communication server 300 may recognize a speakeramong users of the other information processing device 500, 700 andcontents of a speech by a speaker among the users from the speech fromthe other information processing device 500, 700. Furthermore, thecommunication server 300 may recognize actions of the users of the otherinformation processing device 500, 700 from the operation signal fromthe other information processing device 500, 700.

Further, the information processing device 200 may transmit contents tothe other information processing device 500, 700 to share the contentsamong users.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

The invention claimed is:
 1. An information processing apparatuscomprising: a processing circuit configured to (i) recognize a user in acaptured image, (ii) establish a network connection, (iii) generatedisplay information; (iv) receive content image data of a content image,and (iv) transmit, via the network connection, the display informationfor display on a display; and wherein the processing circuit isconfigured, when the network connection is established and at least oneuser in the captured image is recognized, to transmit, via the networkconnection, the display information, in which the display informationincludes the content image data and user image data of the capturedimage, for display as a plurality of display screens displayed one afteranother on the display such that the content image is shifted from afirst side of the display to a second side of the display opposite thefirst side so as to appear on the display, and a user imagecorresponding to the user image data is shifted to the second side ofthe display to fade out of the display.
 2. The information processingapparatus of claim 1, wherein the display information includes secondimage data corresponding to an image indicating the network connection,and wherein the image indicating the network connection is displayedwith the content image on the display after the user image is faded outof the display.
 3. The information processing apparatus of claim 2,wherein the image indicating the network connection is displayed at thesecond side of the display superimposed on the content image.
 4. Theinformation processing apparatus of claim 3, wherein the imageindicating the network connection is displayed at least partiallytransparent.
 5. The information processing apparatus of claim 1, whereinthe captured image is captured at the information processing apparatus.6. The information processing apparatus of claim 1, wherein a firstdisplay screen of the plurality of display screens includes a firstimage portion of the content image which is less than an entirety of thecontent image and a second image portion of the user image which is lessthan an entirety of the user image, wherein a second display screen ofthe plurality of display screens includes a third image portion of thecontent image which is an entirety of the content image and a fourthimage portion indicating the network connection, and wherein the secondscreen is displayed on the display after the first display screen isdisplayed.
 7. The information processing apparatus of claim 6, wherein asize of the first image portion is less than a size of the second imageportion, and the first image portion and the second image portion arearranged in a direction across the display, and wherein the seconddisplay screen does not include any portion of the user image.
 8. Theinformation processing apparatus of claim 6, wherein the first imageportion extends from the first side of the display and not the secondside of the display, and the second image portion extends from thesecond side of the display and not the first side of the display.
 9. Theinformation processing apparatus of claim 6, wherein a third displayscreen of the plurality of display screens includes a fifth imageportion of the content image which is less than an entirety of thecontent image and a sixth image portion of the user image which is lessthan an entirety of the user image, wherein a size of the fifth imageportion is greater than a size of the first image portion, a size of thesixth image portion is less than a size of the second image portion, andthe size of the first image portion is less than the size of the secondimage portion, wherein the third display screen is displayed with thefifth image portion and the sixth image portion arranged in a directionacross the display, in which the fifth image portion extends from thefirst side of the display and not the second side of the display and thesixth image portion extends from the second side of the display and notthe first side of the display, and wherein the third display screen isdisplayed after the first display screen is displayed and before thesecond display screen displayed.
 10. The information processingapparatus of claim 9, wherein the size of the fifth image portion isgreater or less than the size of the sixth image portion.
 11. Theinformation processing apparatus of claim 1, wherein the at least oneuser is recognized in the captured image based on face detection. 12.The information processing apparatus of claim 1 further comprising: animage capture unit to capture the captured image.
 13. An informationprocessing method comprising: controlling, by a processing device, (i)recognizing a user in a captured image, (ii) establishing a networkconnection, (iii) generating display information; (iv) receiving contentimage data of a content image, and (iv) transmitting, via the networkconnection, the display information for display on a display; andcontrolling, by the processing device, when the network connection isestablished and at least one user in the captured image is recognized,transmitting, via the network connection, the display information, inwhich the display information includes the content image data and userimage data of the captured image, for display as a plurality of displayscreens displayed one after another on the display such that the contentimage is shifted from a first side of the display to a second side ofthe display opposite the first side so as to appear on the display, anda user image corresponding to the user image data is shifted to thesecond side of the display to fade out of the display.
 14. Theinformation processing apparatus of claim 1, wherein the networkconnection is established with and transmitted to an external apparatus.15. An information processing method comprising of claim 13, wherein thenetwork connection is established with and transmitted to an externalapparatus.